Dopamine signals for reward value and risk: basic and recent data.

BACKGROUND: Previous lesion, electrical self-stimulation and drug addiction studies suggest that the midbrain dopamine systems are parts of the reward system of the brain. This review provides an updated overview about the basic signals of dopamine neurons to environmental stimuli. METHODS: The described experiments used standard behavioral and neurophysiological methods to record the activity of single dopamine neurons in awake monkeys during specific behavioral tasks. RESULTS: Dopamine neurons show phasic activations to external stimuli. The signal reflects reward, physical salience, risk and punishment, in descending order of fractions of responding neurons. Expected reward value is a key decision variable for economic choices. The reward response codes reward value, probability and their summed product, expected value. The neurons code reward value as it differs from prediction, thus fulfilling the basic requirement for a bidirectional prediction error teaching signal postulated by learning theory. This response is scaled in units of standard deviation. By contrast, relatively few dopamine neurons show the phasic activation following punishers and conditioned aversive stimuli, suggesting a lack of relationship of the reward response to general attention and arousal. Large proportions of dopamine neurons are also activated by intense, physically salient stimuli. This response is enhanced when the stimuli are novel; it appears to be distinct from the reward value signal. Dopamine neurons show also unspecific activations to non-rewarding stimuli that are possibly due to generalization by similar stimuli and pseudoconditioning by primary rewards. These activations are shorter than reward responses and are often followed by depression of activity. A separate, slower dopamine signal informs about risk, another important decision variable. The prediction error response occurs only with reward; it is scaled by the risk of predicted reward. CONCLUSIONS: Neurophysiological studies reveal phasic dopamine signals that transmit information related predominantly but not exclusively to reward. Although not being entirely homogeneous, the dopamine signal is more restricted and stereotyped than neuronal activity in most other brain structures involved in goal directed behavior.RIGHTS : This article is licensed under the BioMed Central licence at http://www.biomedcentral.com/about/license which is similar to the 'Creative Commons Attribution Licence'. In brief you may : copy, distribute, and display the work; make derivative works; or make commercial use of the work - under the following conditions: the original author must be given credit; for any reuse or distribution, it must be made clear to others what the license terms of this work are

Potential Vulnerabilities of Neuronal Reward, Risk, and Decision Mechanisms to Addictive Drugs

How do addictive drugs hijack the brain's reward system? This review speculates how normal, physiological reward processes may be affected by addictive drugs. Addictive drugs affect acute responses and plasticity in dopamine neurons and postsynaptic structures. These effects reduce reward discrimination, increase the effects of reward prediction error signals, and enhance neuronal responses to reward-predicting stimuli, which may contribute to compulsion. Addictive drugs steepen neuronal temporal reward discounting and create temporal myopia that impairs the control of drug taking. Tonically enhanced dopamine levels may disturb working memory mechanisms necessary for assessing background rewards and thus may generate inaccurate neuronal reward predictions. Drug-induced working memory deficits may impair neuronal risk signaling, promote risky behaviors, and facilitate preaddictive drug use. Malfunctioning adaptive reward coding may lead to overvaluation of drug rewards. Many of these malfunctions may result in inadequate neuronal decision mechanisms and lead to choices biased toward drug rewards

Introduction. Neuroeconomics: the promise and the profit

Neuroeconomics investigates the neural mechanisms underlying decisions about rewarding or punishing outcomes (‘economic’ decisions). It combines the knowledge about the behavioural phenomena of economic decisions with the mechanistic explanatory power of neuroscience. Thus, it is about the neurobiological foundations of economic decision making. It is hoped that by ‘opening the box’ we can understand how decisions about gains and losses are directed by the brain of the individual decision maker. Perhaps we can even learn why some decisions are apparently paradoxical or pathological. The knowledge could be used to create situations that avoid suboptimal decisions and harm

Multiple functions of dopamine neurons

Dopamine neurons carry phasic signals for a limited number of behavioural events. The events include, in descending order, reward, physically intense stimuli, risk and punishment. Recent neurophysiological studies have provided interesting details on these functions

Rewarding properties of visual stimuli

The behavioral functions of rewards comprise the induction of learning and approach behavior. Rewards are not only related to vegetative states of hunger, thirst and reproduction but may also consist of visual stimuli. The present experiment tested the reward potential of different types of still and moving pictures in three operant tasks involving key press, touch of computer monitor and choice behavior in a laboratory environment. We found that all tested visual stimuli induced approach behavior in all three tasks, and that action movies sustained consistently higher rates of responding compared to changing still pictures, which were more effective than constant still pictures. These results demonstrate that visual stimuli can serve as positive reinforcers for operant reactions of animals in controlled laboratory settings. In particular, the coherently animated visual stimuli of movies have considerable reward potential. These observations would allow similar forms of visual rewards to be used for neurophysiological investigations of mechanisms related to non-vegetative reward

Herding and Social Pressure in Trading Tasks: A Behavioural Analysis

We extend the experimental literature on Bayesian herding using evidence from a financial decision-making experiment. We identify significant propensities to herd increasing with the degree of herd-consensus. We test various herding models to capture the differential impacts of Bayesian-style thinking versus behavioural factors. We find statistically significant associations between herding and individual characteristics such as age and personality traits. Overall, our evidence is consistent with explanations of herding as the outcome of social and behavioural factors. Suggestions for further research are outlined and include verifying these findings and identifying the neurological correlates of propensities to herd

Choice mechanisms for past, temporally extended outcomes.

Accurate retrospection is critical in many decision scenarios ranging from investment banking to hedonic psychology. A notoriously difficult case is to integrate previously perceived values over the duration of an experience. Failure in retrospective evaluation leads to suboptimal outcome when previous experiences are under consideration for revisit. A biologically plausible mechanism underlying evaluation of temporally extended outcomes is leaky integration of evidence. The leaky integrator favours positive temporal contrasts, in turn leading to undue emphasis on recency. To investigate choice mechanisms underlying suboptimal outcome based on retrospective evaluation, we used computational and behavioural techniques to model choice between perceived extended outcomes with different temporal profiles. Second-price auctions served to establish the perceived values of virtual coins offered sequentially to humans in a rapid monetary gambling task. Results show that lesser-valued options involving successive growth were systematically preferred to better options with declining temporal profiles. The disadvantageous inclination towards persistent growth was mitigated in some individuals in whom a longer time constant of the leaky integrator resulted in fewer violations of dominance. These results demonstrate how focusing on immediate gains is less beneficial than considering longer perspectives.This research was supported by the Wellcome Trust Grants 095495 and 093270 and European Research Council Advanced Grant ERC-2011-AdG 293549.This is the final version. It was first published by Royal Society Publishing at http://rspb.royalsocietypublishing.org/content/282/1810/20141766

Scaling prediction errors to reward variability benefits error-driven learning in humans.

Effective error-driven learning requires individuals to adapt learning to environmental reward variability. The adaptive mechanism may involve decays in learning rate across subsequent trials, as shown previously, and rescaling of reward prediction errors. The present study investigated the influence of prediction error scaling and, in particular, the consequences for learning performance. Participants explicitly predicted reward magnitudes that were drawn from different probability distributions with specific standard deviations. By fitting the data with reinforcement learning models, we found scaling of prediction errors, in addition to the learning rate decay shown previously. Importantly, the prediction error scaling was closely related to learning performance, defined as accuracy in predicting the mean of reward distributions, across individual participants. In addition, participants who scaled prediction errors relative to standard deviation also presented with more similar performance for different standard deviations, indicating that increases in standard deviation did not substantially decrease "adapters'" accuracy in predicting the means of reward distributions. However, exaggerated scaling beyond the standard deviation resulted in impaired performance. Thus efficient adaptation makes learning more robust to changing variability.This work was supported by the Wellcome Trust and the Niels Stensen Foundation.This is the final version of the article. It first appeared from the American Physiological Society via http://dx.doi.org/10.1152/jn.00483.201